Risk and parameter convergence of logistic regression

The logistic loss is strictly convex and does not attain its infimum; consequently the solutions of logistic regression are in general off at infinity. This work provides a convergence analysis of gradient descent applied to logistic regression under no assumptions on the problem instance. Firstly, the risk is shown to converge at a rate O(ln(t)/t). Secondly, the parameter convergence is characterized along a unique pair of complementary subspaces defined by the problem instance: one subspace along which strong convexity induces parameters to converge at rate O(ln(t)/ √ t), and its orthogonal complement along which separability induces parameters to converge in direction at rate O(ln ln(t)/ ln(t)). 1 Overview Logistic regression is the task of finding a vector w ∈ R which approximately minimizes the (empirical) logistic risk, namely